Closed Bug 273109 Opened 20 years ago Closed 19 years ago

Japanese attachment file name garbled, after fix for Bug 243504 and Bug 263462

Categories

(Thunderbird :: Message Compose Window, defect)

x86
Windows XP
defect
Not set
major

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 274264

People

(Reporter: ahndalsoo, Assigned: jshin1987)

Details

(Keywords: fixed-aviary1.0, regression)

Attachments

(2 files)

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; ja-JP; rv:1.7.5) Gecko/20041108 Firefox/1.0
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; ja-JP; rv:1.7.5) Gecko/20041108 Firefox/1.0

Symptom: Japanese attachment file name gets garbled in the following condition.

Condition:
- Japanese character set used in file system is ShiftJIS (i.e Windows environment).
- The 2nd byte of ShiftJIS character is less than 0x80 (i.e range that overlaps
with ASCII). ShiftJIS characters may contain 0x40-0x7E and 0x80-0xFC in 2nd byte.
- The 2nd byte of ShiftJIS character is escaped. 0x5C is the case.

Related code:
/intl/uconv/src/nsUTF8ConverterService.cpp

nsUTF8ConverterService::ConvertURISpecToUTF8(const nsACString &aSpec, const char
*aCharset, nsACString &aUTF8Spec)
{
-snip-
PRBool written = NS_UnescapeURL(PromiseFlatCString(aSpec).get(), aSpec.Length(),
esc_OnlyNonASCII, unescapedSpec);
-snip-
return ToUTF8(unescapedSpec, aCharset, aUTF8Spec);
}

What is going on:
1. For instance, the URI of attachment file name is
"file:///C:/Documents%20and%20Settings/me/%83f%83X%83N%83g%83b%83v/%83%5C.txt
"
2. Calls AttachmentPrettyName().
3. Calls NS_UnescapeURL() with esc_OnlyNonASCII flag.
4. Does not unescape 0x5C.
5. Calls ToUTF8(). The 2nd byte of ShiftJIS character is still escaped.
6. "5C" shows up in mail composition window. Refer to the attached screenshot.

Reproducible: Always
Steps to Reproduce:
1. Compose a new message.
2. Attach a file with Japanese name that contains some particlar character codes.
3. Displayed attachement file name is garbled.
Actual Results:  
Raw character code shows up like "5C".

Expected Results:  
Readable Japanese characters show up.
Attached image Screenshot
Screenshot of garbled attachement file name.
I can reproduce this bug.
Nominating blocking 1.0
Flags: blocking-aviary1.0?
Flags: blocking-aviary1.0? → blocking-aviary1.0-
This bug should be fixed ASAP. Many people make file with Japanese name
and attach it. Please reconsider blocking 1.0.
Severity: normal → major
Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: blocking-aviary1.0- → blocking-aviary1.0?
Flags: blocking-aviary1.0-L10N?
is this a new problem or something that's always been there?
I think this is a 0.9 regression bug.
jshin can you take a look at this? I don't see any changes in our codebase to
our UTF8 converter since 2003 so I don't understand how this is a recent
regression unless the change is with our escaping code. 

Possible sources could be (although after looking back at that patch, I doubt it)

https://bugzilla.mozilla.org/show_bug.cgi?id=268219

This is new since 0.9. I checked http://bugzilla.mozilla.gr.jp/ that is a local
bugzilla in Japan. They report this issue on 0.9 and 1.0RC1. It appears 0.7.3
works fine. There is no report regarding 0.8. NS_UnescapeURL() is a bit
different between these versions as follows.

0.7.3: ignores esc_OnlyNonASCII flag and unescapes the 2nd byte of a ShiftJIS
character.
0.8: I haven't reviewed code.
1.0RC1 and probably 0.9: handles esc_OnlyNonASCII flag and doesn't unescape the
2nd byte of a ShiftJIS character.
Ahn,
this bug looks like the culprit:

https://bugzilla.mozilla.org/show_bug.cgi?id=243504

That looks pretty suspicious to me. What do you think?
Sorry I didn't mean that bug. I meant this one:

https://bugzilla.mozilla.org/show_bug.cgi?id=263462

which was a I18n change to our unescaping code. I'm pretty sure this is the
culprit. Thoughts?
do any of you have a debug build where you can quickly try backing out: 

https://bugzilla.mozilla.org/show_bug.cgi?id=263462

and confirming that this starts working again? 
alternatively, can someone send me an e-mail with an attachment that has a name
in japanese that contains these particular characters that get corrupted?

I have japanese fonts on my machine, so if you send it to me, I can drag it to
my compose window and hopefully see the corruption. 
Scott, I sent you an email with attachment named in Japanese via web-based Yahoo
mail. Then I noticed that Firefox 1.0 too causes character corruption when I
save attachement with Japanese name to my local disk. At ordinay web site, it is
rare to download files with Japanese name. But at web-based email site, we often
find Japanese attachment name. I will report this unless it is already reported.
I got the e-mail attachment just fine. My problem is that on Windows XP when I
save the file, windows converts it into my native charset so the file on my OS
looks like __.txt. I'm trying to figure out a way to get my Operating system to
save it without losing the japanese characters. 
ok, in about 15 minutes, a test build will be up on the ftp servers:

ftp://ftp.mozilla.org/pub/mozilla.org/thunderbird/test/ThunderbirdSetup.exe

This test build backs out the patch for Bug #263462. 

Can one of you guys try it out and see if that fixes this issue as soon as you
can? If we can confirm that it does we'll look into re-spinning our 1.0 bits
with the back out. Thanks!
Mscott, it works fine!
Works fine on my environemnt too. Thank you!
The issue I mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=273109#c12
appears to be a duplicate of https://bugzilla.mozilla.org/show_bug.cgi?id=252309
we are respinning the final 1.0 bits to pick up the backout cited above. Marking
fixed-aviary1.0.

Leaving open and re-assigning to jshin so he can fix his original change in Bug
#263462 to not break on certain japanese characters.
Assignee: mscott → jshin
Flags: blocking-aviary1.0?
Flags: blocking-aviary1.0-L10N?
Flags: blocking-aviary1.0-L10N+
Flags: blocking-aviary1.0+
Keywords: fixed-aviary1.0
Patch for Bug 263462 is backed out on Thunderbird 1.0 release build.
This probably revive Bug 243504 and Bug 239192 (problem in file attach when "#"
or ";" in file path).
Both bugs seems to be resolved by patch for Bug 243504 in conjunction with patch
for Bug 263462 (Bug 243504 depends on Bug 263462). 
See post in MozillaZine forum. 
 ( http://forums.mozillazine.org/viewtopic.php?p=1048564#1048564 )
Change summary(add description on Bug 243504 and Bug 263462 according to Comment
14 by Scott).
Summary: Japanese attachment file name garbled → Japanese attachment file name garbled, after fix for Bug 243504 and Bug 263462
Attached is a patch to leave ascii symbols like # escaped. Applying both of this
and jshin's patch resolves all of Bug 239192, Bug 243504, Bug 263462 and
Bug 273109.
Perhaps esc_FilePath looks better than the combination of esc_Scheme | esc_Host
| esc_Directory | esc_FileBaseName | esc_FileExtension. I just didn't find 
documentation about it in nsEscape.cpp.
Do Mozilla Suite(Trunk and 1.7branch) have same problem?
Patch of Bug 263462 was check in for them too. but not backout.
Add to list in Bug 274366 (regressions meta-bug)
Comment on attachment 168486 [details] [diff] [review]
Patch to leave ascii symbols escaped

Yes, this should , along with my patch, fix all those things. IDN host name
could be a problem, though. We also have to be careful as to which set of flags
to use.
This is fixed on trunk (bug 274264) and the patch will be fixed in  branches
asas I get approved.
I'm resolving it as a dupe of bug 274264 (because there's a landed patch). No
need to keep two bugs around. 


*** This bug has been marked as a duplicate of 274264 ***
Status: NEW → RESOLVED
Closed: 20 years ago
Resolution: --- → DUPLICATE
This bug is for Thunderbird, and was closed as DUP of bug 274264 which is for
Mozilla App Suite.
But bug 274264 was reopened on 2005/1/21 due to backout of fix for it.
So reopen this bug in order to request blocking1.8rc1 (Thunderbird 1.5 rc1).
Status: RESOLVED → REOPENED
Flags: blocking1.8rc1?
Resolution: DUPLICATE → ---
This bug's problem have revived on Tb 1.5 Beta2. (See bug 274264 comment #27)
This is due to backout of patch for bug 274264 from trunk (See bug 274264
comment #26) 
This is not a bug of Tb. It's Core bug.

*** This bug has been marked as a duplicate of 274264 ***
Status: REOPENED → RESOLVED
Closed: 20 years ago19 years ago
Flags: blocking1.8rc1?
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: